The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi-modality signals. In this work, we harness these traits and present a unified multimodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks using a single model, performing text-based, image-based, and even vision-language simultaneous generation. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified transition matrix. Moreover, we design a mutual attention module with fused embedding layer and a unified objective function to emphasise the inter-modal linkages, which are vital for multi-modality generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.

Text-guided diffusion models have shown superior performance in image/video generation and editing. While few explorations have been performed in 3D scenarios. In this paper, we discuss three fundamental and interesting problems on this topic. First, we equip text-guided diffusion models to achieve $\textbf{3D-consistent generation}$. Specifically, we integrate a NeRF-like neural field to generate low-resolution coarse results for a given camera view. Such results can provide 3D priors as condition information for the following diffusion process. During denoising diffusion, we further enhance the 3D consistency by modeling cross-view correspondences with a novel two-stream (corresponding to two different views) asynchronous diffusion process. Second, we study $\textbf{3D local editing}$ and propose a two-step solution that can generate 360$^{\circ}$ manipulated results by editing an object from a single view. Step 1, we propose to perform 2D local editing by blending the predicted noises. Step 2, we conduct a noise-to-text inversion process that maps 2D blended noises into the view-independent text embedding space. Once the corresponding text embedding is obtained, 360$^{\circ}$ images can be generated. Last but not least, we extend our model to perform \textbf{one-shot novel view synthesis} by fine-tuning on a single image, firstly showing the potential of leveraging text guidance for novel view synthesis. Extensive experiments and various applications show the prowess of our 3DDesigner. The project page is available at https://3ddesigner-diffusion.github.io/.

几乎没有零件分割的目的是仅给出几个带注释的样本，将对象的不同部分分开。由于数据有限的挑战，现有的作品主要集中在学习分类器上，而不是预先训练的功能，无法学习针对零件细分的任务特定功能。在本文中，我们建议在“预训练” - “微调”范式中学习特定于任务的功能。我们进行及时设计以减少预训练任务（即图像生成）与下游任务（即部分分段）之间的差距，以便可以利用生成的GAN先验进行分割。这是通过将零件分割图投影到RGB空间中并在RGB分割图和原始图像之间进行插值来实现的。具体而言，我们设计了一种微调策略，以逐步将图像发生器调整到分割生成器中，在该机构中，生成器的监督通过插值从图像到分割图各不等。此外，我们提出了一个两流体系结构，即一个分割流以生成特定于任务的特征，以及一个图像流以提供空间约束。图像流可以视为自我监管的自动编码器，这使我们的模型能够从大规模的支持图像中受益。总体而言，这项工作是试图通过及时设计来探索一代任务和感知任务之间的内部相关性。广泛的实验表明，我们的模型可以在几个部分分割数据集上实现最新性能。

旨在学习具有少量培训数据的生成模型的数据有效gan（DE-GAN）遇到了生成高质量样本的几个挑战。由于数据增强策略在很大程度上已经减轻了训练的不稳定性，因此如何进一步改善De-Gans的生成性能成为热点。最近，对比学习表明，提高了DE-GAN的合成质量的巨大潜力，但相关原则并未得到很好的探索。在本文中，我们对De-Gans中的不同对比度学习策略进行了比较，并确定（i）当前生成性能的瓶颈是潜在空间的不连续性； （ii）与其他对比的学习策略相比，实例扰动可用于潜在空间连续性，从而为De-Gans带来了重大改进。基于这些观察结果，我们提出了FakeClR，该观察只在扰动的假样品上应用对比度学习，并设计了三种相关的训练技术：与噪声​​相关的潜在增强，多样性吸引的排队和排队的遗忘因素。我们的实验结果表明了几乎没有发电和有限数据的新艺术状态。在多个数据集上，与现有DE-GAN相比，Fakeclr获得了15％以上的FID提高。代码可从https://github.com/iceli1007/fakeclr获得。

最近，在蒙版的图像建模中取得了重大进展，以赶上掩盖语言建模。但是，与NLP中的单词不同，图像的语义分解仍然使视觉和语言之间的掩盖自动编码（MAE）不同。在本文中，我们探讨了单词的潜在视觉类似物，即语义部分，并通过提出语义引导的掩盖策略将语义信息集成到MAE的训练过程中。与广泛采用的随机掩蔽相比，我们的掩蔽策略可以逐渐指导网络学习各种信息，即从部分内部模式到零件之间的关系。特别是，我们通过两个步骤实现这一目标。 1）语义部分学习：我们设计了一种自制的部分学习方法，通过利用和完善基于VIT的编码器的多头注意来获得语义部分。 2）语义引导的MAE（SEMMAE）训练：我们设计了一种掩盖策略，该策略从掩盖每个部分中的一部分贴片到掩盖图像中的一部分（整个）部分。关于各种视觉任务的广泛实验表明，Semmae可以通过集成语义信息来学习更好的图像表示。特别是，Semmae在Imagenet-1k上达到了84.5％的微调精度，这使香草Mae的表现优于1.4％。在语义细分和细粒度的识别任务中，Semmae还带来了重大改进并产生最先进的性能。

Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.

Dataset distillation has emerged as a prominent technique to improve data efficiency when training machine learning models. It encapsulates the knowledge from a large dataset into a smaller synthetic dataset. A model trained on this smaller distilled dataset can attain comparable performance to a model trained on the original training dataset. However, the existing dataset distillation techniques mainly aim at achieving the best trade-off between resource usage efficiency and model utility. The security risks stemming from them have not been explored. This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain. Concretely, we inject triggers into the synthetic data during the distillation procedure rather than during the model training stage, where all previous attacks are performed. We propose two types of backdoor attacks, namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw data at the initial distillation phase, while DOORPING iteratively updates the triggers during the entire distillation procedure. We conduct extensive evaluations on multiple datasets, architectures, and dataset distillation techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack success rate (ASR) scores in some cases, while DOORPING reaches higher ASR scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive ablation study to analyze the factors that may affect the attack performance. Finally, we evaluate multiple defense mechanisms against our backdoor attacks and show that our attacks can practically circumvent these defense mechanisms.

Benefiting from the intrinsic supervision information exploitation capability, contrastive learning has achieved promising performance in the field of deep graph clustering recently. However, we observe that two drawbacks of the positive and negative sample construction mechanisms limit the performance of existing algorithms from further improvement. 1) The quality of positive samples heavily depends on the carefully designed data augmentations, while inappropriate data augmentations would easily lead to the semantic drift and indiscriminative positive samples. 2) The constructed negative samples are not reliable for ignoring important clustering information. To solve these problems, we propose a Cluster-guided Contrastive deep Graph Clustering network (CCGC) by mining the intrinsic supervision information in the high-confidence clustering results. Specifically, instead of conducting complex node or edge perturbation, we construct two views of the graph by designing special Siamese encoders whose weights are not shared between the sibling sub-networks. Then, guided by the high-confidence clustering information, we carefully select and construct the positive samples from the same high-confidence cluster in two views. Moreover, to construct semantic meaningful negative sample pairs, we regard the centers of different high-confidence clusters as negative samples, thus improving the discriminative capability and reliability of the constructed sample pairs. Lastly, we design an objective function to pull close the samples from the same cluster while pushing away those from other clusters by maximizing and minimizing the cross-view cosine similarity between positive and negative samples. Extensive experimental results on six datasets demonstrate the effectiveness of CCGC compared with the existing state-of-the-art algorithms.

As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.

Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.